Skip to content

fix(perf): require usable repeat references and complete metric observations - #22

Merged
plotarmordev merged 1 commit into
mainfrom
fix/reference-qualification-14
Sep 10, 2026
Merged

fix(perf): require usable repeat references and complete metric observations#22
plotarmordev merged 1 commit into
mainfrom
fix/reference-qualification-14

Conversation

@plotarmordev

Copy link
Copy Markdown
Owner

Closes #14. Builds on the descriptive ranges/--reference support from PR #10.

A supplied reference can no longer silently vanish from qualification. Reference-aware changes and drift require complete eligible reference evidence, matched ordered completion counts and complete matching baseline declarations. The output exposes reference model/deployment and declared-match/mismatch/unavailable status; matching declarations do not verify physical restoration or a causal A/B/A experiment. Ineligible/missing evidence stays inspectable with explicit reasons and an ineligible exit, not an automatic two-run fallback.

Reference-aware stream metrics also require all paired lane observations across A/B/A2; a few surviving prefill lanes cannot silently replace the full metric. Other available metrics and the ordinary no-reference path retain their scope. Comparison JSON advances to version 3 for the changed reference eligibility contract. Old execution receipts remain readable and unchanged.

Proposal reconciliation: retained min/max ranges and pooled reference ranges are a coarse descriptive filter. This does not add bootstrap confidence intervals, within_noise significance or the proposed absolute-drift threshold. Those require a separate validated statistical design; observed range separation does not certify significance or equality.

Verified: independent SIX/HELP reviews (both initially blocked partial lane dropout; both approved its fix), exact-head fmt/Clippy/workspace tests/release build; regression failed before the dropout fix and passed afterward; loopback CLI tested valid and undeclared repeat behavior, saved pre-change runs remain readable. No real A/B/A inference run or server restoration verification was performed.

@plotarmordev
plotarmordev changed the base branch from feat/cold-prefill-ladder-15 to main September 10, 2026 07:03
@plotarmordev
plotarmordev merged commit c51a2c9 into main Sep 10, 2026
2 checks passed
@plotarmordev
plotarmordev deleted the fix/reference-qualification-14 branch September 10, 2026 07:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

grill-perf compare: A/B/A with noise floor, and ≥3-rep spread

1 participant